Attribution modelling
1 Case study: KNC
“Let’s show a bird’s eye view of our digital channel performance from this year, first. Here we could have a bar chart on number of online purchases and some advertising data ¬on mobile, paid search, affiliations, and social media activities. And some correlations here as well… What do you think?,” said Levi.
“Perhaps, Prisha will ask about the overall improvement compared to last year, right? So, let’s add the same figures from last year and make a comparison here too,” responded Emma.
Levi and Emma, the marketing science team members at KNC–a European firm that sells trendy and quality make-up and skincare products at affordable prices for students and young professionals– sat round a modern-look, high gloss conference table to discuss what to include in their presentation deck prior to their meeting with Prisha, the chief marketing officer (CMO) of KNC.
“I am sure that during the meeting, she will ask about each channel’s contribution to final purchases. This is quite critical for next year’s budget request. How about drilling down to the individual level data and running some attribution models?,” said Levi.
Emma nodded in agreement and said “We can show the results from the ‘first-touch’ and ‘last-touch’ heuristics, and let her decide which model to use for budget-related decisions. Most brands use the ‘last-touch’ model, anyway.”
After a long day at work, Levi and Emma agreed on what to cover in their presentation for the CMO: an overview of digital channel performance and some detailed results from attribution modelling.
Meeting with the CMO
Well, these were the results from the first-touch attribution model. Now, the results from the last-touch model… This model gives full credit to the last digital channel that consumers interacted with right before making a purchase. According to this attribution model, the least effective digital channel is ‘paid search’. It contributes merely 12 per cent to conversions. The most effective is mobile. It contributes 39 per cent. Affiliate marketing surprisingly accounts for 27 per cent in sales conversions. Social media has a moderate impact–its contribution is 22 per cent. Therefore, we propose to reduce our efforts for paid search campaigns and increase our investments in mobile,” said Emma.
Prisha interrupted the presentation and said “I am a little bit puzzled here, Emma. It is hard to believe that social media channel has lower impact on consumer purchases than affiliates. Most of our targeted consumers are young adults who spend considerable time on social networks. These results are not convincing enough.”
Pointing to the below title of an online article from her laptop screen, Prisha continued: “Look at this”.
“Google won’t use the last-click attribution as the default conversion model. I am not sure what exactly they mean by data modelling but they seem to be right in ditching the last-touch model. It is very clear that assigning full credit for a sale to the last digital touchpoint in consumer’s path to purchase may mislead us.”
Meanwhile, as a recently graduated marketing analyst, Levi opens his laptop, digs up his analytics course content and rediscovers that there are other approaches to attribution modelling.
Inspired by his class materials, Levi responded to Prisha:
“Well, we can use more advanced models – for example Markov chain models– to get a better picture of attribution across all these digital touchpoints. Also, we can use Shapley value-based modelling approach to quantify the effect of a channel removal. What would happen if we turn off a channel? Perhaps, by turning off a particular channel we do not lose much on conversion rates…”
“Excellent suggestions,” said Prisha. “My gut feeling is telling me that the model should assign different weights to all the channels on the customer journey. Why don’t you start right now? I will discuss our digital marketing budget with the board next week. So, we need to be quick on this.”
The R application below will help us address the following questions on the KNC case study:
Taking a path-to-purchase view, which customer touchpoint1 contributes more or less to conversion?
Should KNC give up on paid search and put more effort into mobile marketing?
Which attribution modeling approach should the KNC marketing analysts, Levi and Emma, adopt in the future?
2 Preparation and set-up
Before we roll out the analyses, we need to make sure that all our files are organised, and our R environment is set up. To do so, we follow the steps below:
Create a folder and set working directory to that folder on our device
Install and load the R packages needed for this application
Download the dataset to our working directory
2.1 Create a folder and set a working directory
We create a folder on our computer and name it attribution-modeling. Next, we launch the R Studio and open a new qmd file. We save the qmd file in the attribution-modeling folder that we just created.
2.2 Install and load R packages
Next step is to install the R packages. The below code chunk does this task.
```{r, eval=TRUE, error=FALSE, warning=FALSE, message=FALSE}
if (!require("knitr")) install.packages("knitr", repos = "http://cran.us.r-project.org")
if (!require("kableExtra")) install.packages("kableExtra",repos = "http://cran.us.r-project.org")
if (!require("readxl")) install.packages("readxl",repos = "http://cran.us.r-project.org")
if (!require("dplyr"))install.packages("dplyr", repos = "http://cran.us.r-project.org")
if (!require("ChannelAttribution")) install.packages("ChannelAttribution", repos = "http://cran.us.r-project.org")
if (!require("devtools")) install.packages("devtools",repos = "http://cran.us.r-project.org")
if (!require("reshape2")) install.packages("reshape2",repos = "http://cran.us.r-project.org")
if (!require("ggplot2")) install.packages("ggplot2",repos = "http://cran.us.r-project.org")
if (!require("ggthemes")) install.packages("ggthemes",repos="http://cran.us.r-project.org")
if (!require("ggrepel")) install.packages("ggrepel",repos="http://cran.us.r-project.org")
if (!require("RColorBrewer")) install.packages("RColorBrewer",repos="http://cran.us.r-project.org")
if (!require("markovchain")) install.packages("markovchain",repos="http://cran.us.r-project.org")
```Once the packages are installed to our computer, we should load them to R.
```{r, eval=TRUE, error=FALSE, warning=FALSE, message=FALSE}
library(knitr)
library(kableExtra)
library(readxl)
library(dplyr)
library(ChannelAttribution)
library(devtools)
library(reshape2)
library(ggplot2)
library(ggthemes)
library(ggrepel)
library(RColorBrewer)
library(markovchain)
```Next, we will load the dataset.
2.3 Consumer path-to-purchase data
The dataset of this case study is available on the website of the book. The file is named ‘knc_attribution’, and stored in a csv format. We run the following code chunk to load the dataset to R.
```{r}
# load the dataset
attribution <- read.csv(file = "data/knc_attribution.csv", header = TRUE)
```Looking at the dataset attribution from the upper right corner of the R Studio screen, we see that we are given 105 unique online consumer journeys, of which 89 turned into conversion. This corresponds to 85% of conversion rate. Note that this conversion rate is quite high. We typically observe such high conversion rates around heavy promotional times.
The below table provides information on the variables of the dataset and their descriptions.
| Variable | Description |
|---|---|
| users_id | Unique user ID |
| channeln | Online advertising (i.e., mobile, paid search, affiliate, and social media) exposure path, with channel1 referring to the first exposure, channel2 the second, ..., channeln the nth (maximum number of exposure = 45). |
| user_purchase | Consumer final conversion indicator, which takes value of 1 if a specific consumer made purchase (i.e., converted) at the end of her path, and 0 otherwise. |
| null_purchase | The opposite of 'user_purchase', taking value of 1 if there is no purchase and 0 otherwise. |
After getting a glimpse of the data, we can start running the attribution models.
3 Markov chain approach
3.1 Data preparation
To estimate a Markov chain model, we use the ‘ChannelAttribution’ package in R.
We start by knitting consumer advertising exposures into a desired ‘sequenced path’ format for the analysis.The below code generates ‘conversion sequences’ for all consumers.
```{r}
# Construct conversions sequences for all visitors
attribution[is.na(attribution)] <- " "
attribution$path= paste(attribution$channel1,attribution$channel2,attribution$channel3,attribution$channel4,attribution$channel5,attribution$channel6,attribution$channel7,attribution$channel8,attribution$channel9,attribution$channel10,attribution$channel11,attribution$channel12,attribution$channel13,attribution$channel14,attribution$channel15,attribution$channel16,attribution$channel17,attribution$channel8,attribution$channel9,attribution$channel20,attribution$channel21,attribution$channel22,attribution$channel23 ,attribution$channel24,attribution$channel25,attribution$channel26,attribution$channel27,attribution$channel28,attribution$channel29,attribution$channel30,attribution$channel31,attribution$channel32,attribution$channel33,attribution$channel34,attribution$channel35,attribution$channel36,attribution$channel37,attribution$channel38,attribution$channel39,attribution$channel40, attribution$channel41, attribution$channel42, attribution$channel43, attribution$channel44,attribution$channel45, sep=">")
path <- gsub("\\s.*","", attribution$path)
attribution$cleaned_path <-gsub("\\W$","", path)
```If we take a look at the dataset again, we see that there is a new variable created, called cleaned_path.
Now let’s look at the paths of first five consumers in our dataset:
```{r}
# take a look at the data
data_view <- attribution [1:5,50]
data_view %>%
kable() %>%
kable_styling()
```| x |
|---|
| mobile>paid_search>paid_search>paid_search>affiliate>mobile>paid_search>paid_search>social_media>mobile>social_media>paid_search>social_media>mobile>affiliate>paid_search>paid_search>paid_search>social_media>>>>>>>>>>>>>>>>>>>>>>>>> |
| paid_search>mobile>mobile>paid_search>affiliate>paid_search>mobile>paid_search>mobile>mobile>paid_search>affiliate>affiliate>affiliate>affiliate>paid_search>affiliate>paid_search>mobile>affiliate>affiliate>paid_search>affiliate>>>>>>>>>>>>>>>>>>>>> |
| social_media>social_media>mobile>affiliate>social_media>paid_search>mobile>mobile>affiliate>>>>>>>>>mobile>affiliate>>>>>>>>>>>>>>>>>>>>>>>>> |
| mobile>affiliate>mobile>social_media>affiliate>mobile>social_media>mobile>>>>>>>>>>mobile>>>>>>>>>>>>>>>>>>>>>>>>>> |
| paid_search>paid_search>affiliate>paid_search>mobile>paid_search>paid_search>affiliate>mobile>affiliate>paid_search>affiliate>affiliate>affiliate>affiliate>paid_search>mobile>affiliate>mobile>affiliate>mobile>mobile>paid_search>>>>>>>>>>>>>>>>>>>>> |
We can already observe some heterogeneity across consumers in terms of:
starting touchpoint (i.e., first-touch)
ending touchpoint before (non)conversion (i.e., last-touch), and
composition and length of paths.
For model estimation, we decide to take the first 80 consumers in the dataset as our training set and keep the remaining 25 observations for the test set. The below code chunk splits the data into training and test sets.
```{r}
# define training and testing set
attribution_train <-attribution[1:80,]
attribution_test <-attribution[81:105,]
```As a final preparatory step, we need to create a ‘channel stack’, through which we summarize consumer paths and calculate the total number of conversions (frequency)and non-conversions for each path pattern. The following code chunk will help us generate that channel stack:
```{r}
channel_stack = attribution_train %>%
group_by(cleaned_path) %>%
summarize(conversion = sum(user_purchase),non_conversion = sum(null_purchase)) %>%
collect()
# Take a look at prepared dataset
data_view <- channel_stack [1:10,]
data_view %>%
kable() %>%
kable_styling()
```| cleaned_path | conversion | non_conversion |
|---|---|---|
| affiliate>mobile>affiliate>social_media>social_media>mobile>mobile>mobile>mobile>>>>>>>>>mobile>mobile>>>>>>>>>>>>>>>>>>>>>>>>> | 1 | 0 |
| affiliate>paid_search>affiliate>paid_search>affiliate>affiliate>affiliate>affiliate>affiliate>mobile>affiliate>paid_search>paid_search>mobile>paid_search>paid_search>affiliate>affiliate>affiliate>paid_search>affiliate>mobile>affiliate>paid_search>affiliate>mobile>affiliate>affiliate>paid_search>paid_search>paid_search>mobile>affiliate>affiliate>affiliate>affiliate>paid_search>affiliate>paid_search>affiliate>affiliate>>> | 1 | 0 |
| affiliate>paid_search>affiliate>paid_search>mobile>mobile>paid_search>paid_search>mobile>affiliate>affiliate>paid_search>mobile>paid_search>affiliate>mobile>paid_search>paid_search>mobile>paid_search>affiliate>paid_search>affiliate>social_media>paid_search>mobile>paid_search>affiliate>mobile>affiliate>affiliate>mobile>affiliate>mobile>paid_search>mobile>mobile>mobile>paid_search>affiliate>affiliate>mobile>affiliate>paid_search>mobile | 1 | 0 |
| affiliate>paid_search>affiliate>paid_search>social_media>mobile>social_media>paid_search>social_media>affiliate>social_media>paid_search>mobile>paid_search>social_media>mobile>paid_search>paid_search>social_media>paid_search>affiliate>paid_search>social_media>social_media>social_media>mobile>paid_search>social_media>mobile>affiliate>affiliate>mobile>affiliate>mobile>paid_search>social_media>mobile>mobile>paid_search>affiliate>social_media>mobile>social_media>social_media>mobile | 4 | 0 |
| affiliate>paid_search>affiliate>social_media>social_media>mobile>mobile>paid_search>mobile>>>>>>>>>paid_search>mobile>>>>>>>>>>>>>>>>>>>>>>>>> | 2 | 2 |
| affiliate>paid_search>paid_search>mobile>affiliate>social_media>mobile>paid_search>social_media>mobile>paid_search>mobile>mobile>affiliate>social_media>paid_search>mobile>paid_search>social_media>affiliate>affiliate>mobile>mobile>paid_search>paid_search>paid_search>social_media>mobile>mobile>mobile>mobile>>>>>>>>>>>>> | 5 | 0 |
| mobile>affiliate>mobile>social_media>affiliate>mobile>social_media>mobile>>>>>>>>>>mobile>>>>>>>>>>>>>>>>>>>>>>>>>> | 2 | 0 |
| mobile>mobile>affiliate>paid_search>affiliate>mobile>affiliate>paid_search>paid_search>mobile>>>>>>>>paid_search>paid_search>>>>>>>>>>>>>>>>>>>>>>>>> | 1 | 0 |
| mobile>mobile>social_media>social_media>affiliate>affiliate>paid_search>mobile>social_media>affiliate>affiliate>mobile>paid_search>mobile>social_media>social_media>social_media>mobile>social_media>paid_search>mobile>affiliate>paid_search>mobile>mobile>mobile>paid_search>affiliate>mobile>mobile>paid_search>affiliate>social_media>mobile>mobile>affiliate>paid_search>affiliate>mobile>>>>> | 5 | 0 |
| mobile>paid_search>affiliate>paid_search>mobile>mobile>affiliate>paid_search>paid_search>mobile>>>>>>>>paid_search>paid_search>>>>>>>>>>>>>>>>>>>>>>>>> | 0 | 1 |
Now we are ready to estimate the Markov chain model with the cleaned version of our data.
3.2 Markov chain modelling
To estimate the Markov chain model, we use the markov_model function. Specifically, we estimate a third-order Markov model so that the ‘memory’ of the chain goes back to the most recent three states. Our assumption here is that consumer journeys typically cannot be restricted to the most recent state, i.e., consumers have a longer memory. Recall that first-order Markov model assumes that the current state is only determined by the previous or the most recent state.
```{r}
# Third-order Markov chain model
markov = markov_model(channel_stack, "cleaned_path", "conversion", order=3)
table_markov<-data.frame(channel = markov$channel_name,total_conversions =round(markov$total_conversions), percent = round(markov$total_conversions/sum(attribution_train$user_purchase),3))
table_markov
```
Number of simulations: 100000 - Convergence reached: 0.34% < 5.00%
Percentage of simulated paths that successfully end before maximum number of steps (44) is reached: 89.52%
[1] "*** Install ChannelAttribution Pro for free running install_pro(). Visit https://channelattribution.io for more info. Set flg_pro=FALSE to hide this message."
channel total_conversions percent
1 affiliate 16 0.242
2 mobile 17 0.258
3 social_media 16 0.237
4 paid_search 18 0.263The results based on Markov chain model tells us quite a different story compared to what the KNC analysts found using the last-touch heuristic.
Markov chain model informs us that:
The effectiveness of four touchpoints in driving conversions do not differ dramatically from each other.
Both social media advertising and affiliate advertising contribute around 24% to conversion while both paid search and mobile channels contribute around 26%.
To further make sense of what we have found, we can obtain the transition probabilities between all states through the following codes:
```{r}
# Transition Matrix of order 3
trans_3rd_order<-transition_matrix(attribution_train, var_path = "cleaned_path", var_conv = "user_purchase", var_null = "null_purchase", order=3, sep=">", flg_equal=TRUE)
```[1] "*** Install ChannelAttribution Pro for free running install_pro(). Visit https://channelattribution.io for more info. Set flg_pro=FALSE to hide this message."To summarize the transition matrix of third-order Markov model, we obtain the transition probabilities to conversion state using the three-channel paths.
channel_3rd_order transition_probability
1 social_network>paid_search>affiliate 0.455
2 mobile>social_media>mobile 0.273
3 mobile>mobile>mobile 0.250
4 mobile>mobile>affiliate 0.238
5 paid_search>affiliate>mobile 0.227
6 social_media>affiliate>social_media 0.200
7 social_media>social_media>affiliate 0.190
8 social_media>social_media>mobile 0.161
9 social_media>mobile>affiliate 0.156
10 mobile>mobile>paid_search 0.125
11 mobile>paid_search>paid_search 0.116
12 paid_search>paid_search>mobile 0.100
13 mobile>paid_search>mobile 0.097
14 paid_search>paid_search>social_media 0.090
15 paid_search>mobile>paid_search 0.086For example, we can see that consumers with the last three channel exposures Social Media > Paid Search > Affiliate has the highest conversion probability (p=0.455).
We can imagine that such a path might highly describe a teenage consumer behaviour online. They are attracted by fancy social network advertising on Instagram or Facebook. Then, they click on paid search advertising to get further information about the product.Later, they get further enticed by repeatedly browsing on affiliate websites, and finally decide to purchase the latest beauty product of KNC.
Question: Considering the Markov transition matrix and path-to-conversion table above, can you try to infer why we have such a different conclusion about social media and affiliate channel contributions based on Markov chain and last-touch modelling approaches?
3.3 Visualisation
Markov graph
A Markov graph helps us further understand the paths that consumers in our dataset have taken towards conversion and non-conversion. We could plot Markov graph for the third-order Markov chain model that we just estimated. However, due to complexity in visual representation of the third-order model, here we obtain a graph of the first-order Markov chain model to illustrate how to interpret such graphs from Markov chain models.
Running the code chunks below gets us a nice-looking Markov graph.
```{r}
# Estimate a first-order Markov Model to plot Markov graph for illustration
trans_1st<-transition_matrix(attribution_train, var_path = "cleaned_path", var_conv = "user_purchase", var_null = "null_purchase", order=1, sep=">", flg_equal=TRUE)
trans_matrix_1st <-trans_1st$transition_matrix
df_dummy <- data.frame(channel_from = c('(start)', '(conversion)', '(null)'),channel_to = c('(start)', '(conversion)', '(null)'), transition_probability = c(0,1,1))
trans_matrix_1st <-rbind(trans_matrix_1st, df_dummy)
trans_matrix_1st$channel_from <-factor(trans_matrix_1st$channel_from, levels = c('(start)', '(conversion)', '(null)', '1','2','3','4'))
trans_matrix_1st$channel_to <-factor(trans_matrix_1st$channel_to, levels =c('(start)', '(conversion)', '(null)', '1','2','3','4'))
trans_matrix_1st <-dcast(trans_matrix_1st, channel_from~channel_to, value.var = 'transition_probability')
trans_matrix_final <- matrix(data = as.matrix(trans_matrix_1st[, -1]), nrow = nrow(trans_matrix_1st[, -1]), ncol=ncol(trans_matrix_1st[,-1]), dimnames = list(c(as.character(trans_matrix_1st[,1])), c(colnames(trans_matrix_1st[,-1]))))
trans_matrix_final[is.na(trans_matrix_final)] <-0
trans_matrix1 <- new("markovchain", transitionMatrix = trans_matrix_final)
```[1] "*** Install ChannelAttribution Pro for free running install_pro(). Visit https://channelattribution.io for more info. Set flg_pro=FALSE to hide this message."```{r fig, fig.height = 15, fig.width = 10, fig.align = "center"}
# Plot the Markov graph
plot(trans_matrix1, edge.arrow.siz=0.01, main="Markov Graph", fill=c("yellow") )
```
On the Markov graph above:
Nodes 1, 2, 3, and 4 refer to mobile, paid search, affiliate, and social media advertising, respectively.
Nodes ‘conversion’ and ‘null’ refer to conversion and non-conversion, respectively.
Node ‘start’ represents a starting point of all paths, as is requested by the plotting package that we use.
Arrows point from the ‘departing’ node (i.e.,\(({n-1})^{th}\) channel) to the ‘arriving’ node (i.e., \({n}^{th}\) channel).
The numeric value along an arrow pointing from node \(i\) to node \(j\) is the transition probability of a customer first getting exposed to channel \(i\) and then to channel \(j\).
Looking at the graph, the probability of purchase for a consumer with a simple path of ‘start > paid_search > conversion’ is \(0.41 * 0.03 = 0.0123\), since the transition probability from start to paid search is 0.41, and that from paid search to conversion is 0.03.
Transition heatmap
We may also want to create a heatmap for the transition matrix of our estimated Markov model using the following codes. Note that this map is also created based on first-order markov model results (trans_1st), since graphical representation of the third-order Markov transition heatmap is too densed and difficult to read.
```{r}
df_plot_trans <-trans_1st$transition_matrix
# If you prefer a map with 3rd order Markov chain, you may simply run the following line of code instead of the one above:
#df_plot_trans <-trans_3rd_order$transition_matrix
# Start setting up plotting index
cols <- c("#e7f0fa", "#c9e2f6", "#95cbee", "#0099dc", "#4ab04a", "#ffd73e", "#eec73a",
"#e29421")
t <- max(trans_1st$transition_matrix$transition_probability)
ggplot(df_plot_trans, aes(y = channel_from, x = channel_to, fill = transition_probability)) +
theme_minimal() +
geom_tile(colour = "white", width = .9, height = .9) +
scale_fill_gradientn(colours = cols, limits = c(0, t),
breaks = seq(0, t, by = t/4),
labels = c("0", round(t/4*1, 2), round(t/4*2, 2), round(t/4*3, 2), round(t/4*4, 2)),
guide = guide_colourbar(ticks = T, nbin = 50, barheight = .5, label = T, barwidth = 10)) +
geom_text(aes(label = round(transition_probability, 2)), fontface = "bold", size = 4) +
theme(legend.position = 'bottom',
legend.direction = "horizontal",
panel.grid.major = element_blank(),
panel.grid.minor = element_blank(),
plot.title = element_text(size = 14, face = "bold", vjust = 2, color = 'black', lineheight = 0.8),
axis.title.x = element_text(size = 12, face = "bold"),
axis.title.y = element_text(size = 12, face = "bold"),
axis.text.y = element_text(size = 12, face = "bold", color = 'black'),
axis.text.x = element_text(size = 12, angle = 90, hjust = 0.5, vjust = 0.5, face = "plain")) +
ggtitle("Transition matrix heatmap")
```
In the transition matrix heatmap, the rows represent the ‘starting point’ and the columns represent the ‘ending point’ of each one-step transition.From this heatmap, we can reach the same conclusions as with the Markov graph.
Note that there are no rows representing conversion or null states on the matrix because in our setting there are no paths starting from a conversion or starting from a non-conversion. That is also why the cells referring to the transition from start to conversion and null are empty.
4 Model comparison
We would like to see the difference in outcomes of the traditional heuristic approaches (i.e., first-touch, last-touch) and the model-based approach (i.e., Markov chain).
We first use heuristic_models function to quickly obtain results for the heuristic models:
```{r warning=FALSE}
# Heuristic Models
heuristic_models(channel_stack, "cleaned_path", "conversion", NULL, sep = ">")
```[1] "*** Install ChannelAttribution Pro for free! Run install_pro(). Set flg_pro=FALSE to hide this message."
channel_name first_touch last_touch linear_touch
1 affiliate 14 23 14.80225
2 mobile 17 24 19.23514
3 social_media 8 4 14.13698
4 paid_search 28 16 18.82563Note that the output of heuristic_function also includes the linear touch model. As explained previously in the chapter, linear (even-split) touch model is a simple heuristic model that assumes that each channel contributes to final conversion equally. We do not focus on this specific model in this chapter as it is not of interest to KNC.
Now we summarize model results of all three models into one table:
```{r}
table_summary <- data.frame(Item = c("Mobile" ,"Paid Search", "Affiliate", "Social Media"), First_Touch = round(c(17, 28, 14, 8)/67,3), Last_Touch= round(c(26, 15, 18, 8)/67,3), Markov_Chain = c(0.260, 0.245, 0.238, 0.257))
table_summary
``` Item First_Touch Last_Touch Markov_Chain
1 Mobile 0.254 0.388 0.260
2 Paid Search 0.418 0.224 0.245
3 Affiliate 0.209 0.269 0.238
4 Social Media 0.119 0.119 0.257The table above suggests that one can get to very different conclusions by adopting first-touch, last-touch, and Markov chain approach, respectively. Such difference is even clearer if we contrast model results by plotting a bar chart:
```{r}
# create a dataset
method <- c(rep("First Touch",4) ,rep("Last Touch",4) ,rep("Markov chain",4))
channel <- rep(c("Mobile" , "Paid Search" , "Affiliate", "Social Media") , 3)
value <- c(0.254, 0.418, 0.209, 0.119, 0.388, 0.224, 0.269, 0.119,0.260,0.245, 0.238, 0.257)
data <- data.frame(method,channel,value)
# Stacked + percent
ggplot(data, aes(fill=channel, y=value, x=method)) +
geom_bar(position="fill", stat="identity")
```
While first-touch model guides KNC to invest more in paid search and less in social media, last-touch model would suggest strong influence of mobile advertising. Finally, Markov graph approach suggests that the contribution made by all four touchpoints are actually quite evenly distributed.
Which model should KNC choose?
There are two options available for KNC when deciding which model to choose:
KNC should try to make sense of these results, first. Prisha can combine the model-based evidence with her managerial intuition. Given her concerns about the touch-based models, Prisha is advised to make her next budget allocation decisions based on Markov chain model output. At the end of the campaign period, she can assess whether there is a significant improvement in conversion rates.
Alternative approach would be to assess the predictive power of attribution models by using a larger dataset. Research on attribution models demonstrate that Markov chain models provide a fairer allocation of weights to channels and perform better than heuristic (e.g., first-touch and last-touch) methods in predicting conversion rates because they take into account the interplay across touchpoints and sequentiality in a customer journey (Anders et al. 2016).In appendix, we illustrate how predictive performance of these models can be assessed using the current dataset. In our illustration, Markov model performs better than heuristic methods. However, we take those results with a grain of salt because some paths to purchase occur only a few times in the test set, which may make conversion probabilities less reliable.
What if a particular touchpoint is removed?
What would have happened to conversion rate when a particular channel was removed from path-to-purchase? To address this question, we will use Shapley value-based approach.
5 Shapley value-based approach
This modelling approach allows us to evaluate the relative importance of a customer touchpoint to conversion.
Using mobile marketing as an example, we will calculate the drop in conversion probability. The scale of the drop will be the importance of mobile marketing in converting customers. Specifically, we will calculate the following:
\[ (No. of conversions/No. of observations) - (No. of conversions_{nomobile}/No. of observations_{nomobile}) \] We do the same for each of the four touchpoints in the dataset.
The below code chunk searches if each of the paths contains mobile, paid search, affiliate, and social media or not. Then, it generates four indicator variables.
```{r}
# Search if each of the paths contains mobile, paid search, affiliate, and social media or not, and generate four indicator variables.
channel_stack$with_mobile = grepl(x=channel_stack$cleaned_path,pattern="mobile")
channel_stack$with_paid_search = grepl(x=channel_stack$cleaned_path,pattern="paid_search")
channel_stack$with_affiliate = grepl(x=channel_stack$cleaned_path,pattern="affiliate")
channel_stack$with_social_media = grepl(x=channel_stack$cleaned_path,pattern="social_media")
channel_stack$no_mobile_conversion<-ifelse (channel_stack$with_mobile=="FALSE", channel_stack$conversion,0)
channel_stack$no_mobile_count<-ifelse (channel_stack$with_mobile=="FALSE",channel_stack$conversion+ channel_stack$non_conversion,0)
channel_stack$no_paidsearch_conversion<-ifelse (channel_stack$with_paid_search=="FALSE", channel_stack$conversion,0)
channel_stack$no_paidsearch_count<-ifelse (channel_stack$with_paid_search=="FALSE", channel_stack$conversion+ channel_stack$non_conversion,0)
channel_stack$no_affiliate_conversion<-ifelse (channel_stack$with_affiliate=="FALSE", channel_stack$conversion,0)
channel_stack$no_affiliate_count<-ifelse (channel_stack$with_affiliate=="FALSE", channel_stack$conversion+ channel_stack$non_conversion,0)
channel_stack$no_social_media_conversion<-ifelse (channel_stack$with_social_media=="FALSE", channel_stack$conversion,0)
channel_stack$no_social_media_count<-ifelse (channel_stack$with_social_media=="FALSE", channel_stack$conversion+ channel_stack$non_conversion,0)
```Next, we compute the Shapley values for each customer touchpoint and convert them to percentage terms.
```{r}
# get Shapley values for each touchpoint
shapley_mobile<-(sum(channel_stack$conversion)/105)-(67-sum(channel_stack$no_mobile_conversion))/(105-sum(channel_stack$no_mobile_count))
shapley_paidsearch<-(sum(channel_stack$conversion)/105)-(67-sum(channel_stack$no_paidsearch_conversion))/(105-sum(channel_stack$no_paidsearch_count))
shapley_affiliate<-sum(channel_stack$conversion)/105-(67-sum(channel_stack$no_affiliate_conversion))/(105-sum(channel_stack$no_affiliate_count))
shapley_socialmedia<-sum(channel_stack$conversion)/105-(67-sum(channel_stack$no_social_media_conversion))/(105-sum(channel_stack$no_social_media_count))
shapley_mobile
shapley_paidsearch
shapley_affiliate
shapley_socialmedia
```[1] 0.01064426
[1] 0.01064426
[1] 0.003479853
[1] 0.008095238```{r}
# converting Shapley values to percentage terms
shapley_mobile_pct<-shapley_mobile/(shapley_mobile+shapley_paidsearch+shapley_affiliate +shapley_socialmedia)
shapley_paidsearch_pct <-shapley_paidsearch/(shapley_mobile+shapley_paidsearch+shapley_affiliate +shapley_socialmedia)
shapley_affiliate_pct<- shapley_affiliate/(shapley_mobile+shapley_paidsearch+shapley_affiliate +shapley_socialmedia)
shapley_socialmedia_pct<- shapley_socialmedia/(shapley_mobile+shapley_paidsearch+shapley_affiliate+shapley_socialmedia)
shapley_mobile_pct
shapley_paidsearch_pct
shapley_affiliate_pct
shapley_socialmedia_pct
```[1] 0.3238919
[1] 0.3238919
[1] 0.1058878
[1] 0.2463284The output from the Shapley model suggests that the impact of mobile and paid search advertising is the largest because they are associated with the largest drop in conversion percentage, followed by social media, and then affiliate. Thus, removing mobile and paid search channels would not be an appropriate action for KNC. Our further advice to Prisha is that she needs to evaluate the strategic implications of a channel removal very carefully with a forward-looking perspective because consumers may be pushed to other touchpoints (e.g. paid search), which may result in a higher cost to KNC. Finally, conclusions from Shapley model are not causal. To understand the causal impact of a channel removal, randomized field experiments should be adopted, in which consumers are randomly assigned to control and treatment groups.
6 Conclusions
Overall, the marketing group at KNC seems to be rather confident of the following decisions to make:
KNC should analyze and predict consumer conversion by switching from first- and last-touch heuristic approaches to Markov chain model.
Instead of downgrading their emphasis on affiliate and social media advertising, KNC should keep investing in all of the customer touchpoints.
KNC can use the Shapley value-based modelling approach to quantify the potential impact of a channel removal on consumers’ purchases. However, removal of a channel (e.g., social media) may result in significant changes in the customer journey. Consumers may be pushed to other touchpoints (e.g. paid search) that might be more costly to the firm. Therefore, findings from Shapley model should always be combined with a careful strategic analysis. For instance, even though the model finds zero contribution for the social media channel, the manager may still want to be present on social media to increase the brand awareness in the market.
These results do not imply causality. To understand the causal impact of a touchpoint on the relevant performance metrics, KNC is suggested to run randomized field experiments.
7 Appendix
7.1 Predictive power of attribution models
Here we will examine if Markov chain model outperforms the other two heuristics models (i.e., first-touch and last-touch) in terms of predictive accuracy. It is important to mention that we do not include the Shapley value-based in the prediction exercise because the model essentially explores ‘what-if’ scenarios.
To make comparisons on the predictive power of the aforementioned attribution models, we pull out the last 25 observations from the dataset and use them as a test set, attribution_test.
First-touch model
For the first-touch model, the idea of prediction is quite straightforward. We need extract the first channel exposure of each consumer in the test set and refer back to what we obtained from the training set about conversion probability of each channel type. Depending on what the first touch of each consumer in the test set is, we can simply match the corresponding estimated probability of conversion as predicted outcome.
```{r warning=FALSE}
# recall estimation results from First Touch model
table_summary2 <- data.frame(Item = c("Mobile" ,"Paid Search", "Affiliate", "Social Media"), First_Touch = round(c(17, 28, 14, 8)/67,3))
table_summary2
``` Item First_Touch
1 Mobile 0.254
2 Paid Search 0.418
3 Affiliate 0.209
4 Social Media 0.119First-touch model prediction
```{r warning=FALSE}
# Get the first channel exposure of each consumer in the test set
attribution_test$first_touch <-attribution_test$channel1
# Predict based on training set result
attribution_test$first_touch_prediction <-ifelse(attribution_test$first_touch=="mobile", 0.254, ifelse(attribution_test$first_touch=="paid_search",0.418, ifelse(attribution_test$first_touch=="affiliate", 0.209,0.119)))
# Compare prediction with actual purchase based on first-touch model
first_touch<-data.frame(user_id = attribution_test$users_id, first_touch = attribution_test$first_touch, user_purchase = attribution_test$user_purchase, predicted_conversion_probability = attribution_test$first_touch_prediction)
first_touch
``` user_id first_touch user_purchase predicted_conversion_probability
1 81 mobile 1 0.254
2 82 mobile 1 0.254
3 83 paid_search 1 0.418
4 84 affiliate 1 0.209
5 85 social_media 1 0.119
6 86 affiliate 0 0.209
7 87 paid_search 0 0.418
8 88 paid_search 0 0.418
9 89 mobile 0 0.254
10 90 paid_search 1 0.418
11 91 mobile 0 0.254
12 92 paid_search 0 0.418
13 93 paid_search 1 0.418
14 94 paid_search 1 0.418
15 95 social_media 1 0.119
16 96 mobile 1 0.254
17 97 social_media 1 0.119
18 98 affiliate 1 0.209
19 99 affiliate 0 0.209
20 100 mobile 0 0.254
21 101 affiliate 0 0.209
22 102 paid_search 1 0.418
23 103 paid_search 1 0.418
24 104 social_media 1 0.119
25 105 mobile 1 0.254Here we spot some issues with predictions from the first-touch model. For example, for consumers 87 and 88, the predicted conversion probabilities are high (0.418 for both consumers). However, when we look at the actual data, we see that they did not purchase at all.
Last-touch model
We follow the same way as we did for the first-touch model above.
```{r}
table_summary3 <- data.frame(Item = c("Mobile" ,"Paid Search", "Affiliate", "Social Media"), Last_Touch = round(c(26, 15, 18, 8)/67,3))
table_summary3
``` Item Last_Touch
1 Mobile 0.388
2 Paid Search 0.224
3 Affiliate 0.269
4 Social Media 0.119Last-touch prediction
```{r}
for(i in 1:25){
s <- 45-sum(attribution_test[i,]==" ")
attribution_test$channel_n[i] <- attribution_test[i,s+1]}
# Get the last channel exposure of each consumer in the testing set
attribution_test$last_touch <-attribution_test$channel_n
# Predict based on training set result
attribution_test$last_touch_prediction <-ifelse(attribution_test$last_touch=="mobile", 0.388, ifelse(attribution_test$last_touch=="paid_search",0.224, ifelse(attribution_test$last_touch=="affiliate", 0.269,0.119)))
# Compare prediction with actual purchase based on last-touch model
last_touch<-data.frame(user_id = attribution_test$users_id, user_purchase = attribution_test$user_purchase, predicted_conversion_probability = attribution_test$last_touch_prediction)
last_touch
``` user_id user_purchase predicted_conversion_probability
1 81 1 0.119
2 82 1 0.119
3 83 1 0.119
4 84 1 0.269
5 85 1 0.119
6 86 0 0.119
7 87 0 0.119
8 88 0 0.119
9 89 0 0.119
10 90 1 0.119
11 91 0 0.119
12 92 0 0.119
13 93 1 0.119
14 94 1 0.119
15 95 1 0.119
16 96 1 0.119
17 97 1 0.119
18 98 1 0.119
19 99 0 0.119
20 100 0 0.119
21 101 0 0.119
22 102 1 0.119
23 103 1 0.119
24 104 1 0.119
25 105 1 0.119From these results, we understand that the last-touch model is able to predict purchases to a certain extent, but definitely not perfectly.
Markov chain model
To get predictions using Markov model, we refer to the third-order Markov transition matrix. Similar to the rationale in first-touch model prediction:
We extract all the paths that lead to conversion and retrieve their corresponding transition probabilities to conversion.
We then match these paths with the last three channel exposures of each customer in the test set to obtain the predictions.
The code below is to see the number of conversions in the test set.
```{r}
# take a look at total number of conversions in the test set out of 25 observations.
testing_conversion <- sum(attribution_test$user_purchase)
testing_conversion
```[1] 16We understand that in the test set there are 16 conversions out of 25 observations. Next, we move to Markov model predictions.
Markov model prediction
We create a variable last_three_channels in the test set, representing the last three touchpoints that each consumer interacted with along their journey.
```{r warning=FALSE}
for(i in 1:25){
s <- 45-sum(attribution_test[i,]==" ")
attribution_test$channel_n[i] <- attribution_test[i,s+1]
attribution_test$channel_n_1[i] <- attribution_test[i,s]
attribution_test$channel_n_2[i] <- attribution_test[i,s-1]
attribution_test$last_three_channels[i] <- paste(attribution_test$channel_n_2[i], attribution_test$channel_n_1[i], attribution_test$channel_n[i], sep = ">")
}
```Now we simply need to match the last three exposures of consumers in the test set with their corresponding conversion probability, if any.
```{r}
# Matching Markov model estimation and actual data in the test set to create predictions
merged_attribution <- merge(attribution_test, markov_result, by.x ="last_three_channels" , by.y = "channel_3rd_order", all.x = TRUE, all.y=FALSE)
```After matching, we can summarize the results on actual vs. predicted values by listing customers in the test set together with their actual conversion (user_purchase) and predicted conversion probability (predicted_conversion_probability), and order the new dataset by predicted probability in descending order.
```{r}
performance <-data.frame(users_id = merged_attribution$users_id, user_purchase = merged_attribution$user_purchase, predicted_conversion_probability = merged_attribution$transition_probability)
performance[is.na(performance)] <- 0
markov_prediction <- performance[order(-performance$predicted_conversion_probability),]
markov_prediction
``` users_id user_purchase predicted_conversion_probability
1 81 1 0
2 82 1 0
3 83 1 0
4 85 1 0
5 86 0 0
6 87 0 0
7 88 0 0
8 89 0 0
9 90 1 0
10 91 0 0
11 92 0 0
12 93 1 0
13 94 1 0
14 95 1 0
15 96 1 0
16 97 1 0
17 98 1 0
18 100 0 0
19 101 0 0
20 102 1 0
21 103 1 0
22 104 1 0
23 105 1 0
24 84 1 0
25 99 0 07.2 Predictive performance comparison across models
To determine which model performs better in predicting consumer conversion, we compare predicted conversion probabilities for each consumer with the actual conversion data. Specifically, we need to compare model performance in terms of their accuracy and efficiency in identifying consumers with high likelihood of conversion.
To do this, we need to take the following steps:
Rank data by predicted conversion probability (i.e., predicted_conversion_probability) in descending order. Note that an observation with non-conversion (i.e., user_purchase=0) might correspond to a positive of even high predicted probability of conversion. This tends to happen if our prediction model performs poorly.
Starting from the first observation of ranked data, count the cumulative number of actual conversions. The ‘quicker’ the cumulative count reaches total number of conversion (16 in this case), the more capable the model is in identifying promising consumers through their channel exposure paths.
Let us start with the first-touch model.
First-touch model
```{r}
# sort data in descending order
first_touch_prediction <- first_touch[order(-first_touch$predicted_conversion_probability),]
first_touch_prediction
``` user_id first_touch user_purchase predicted_conversion_probability
3 83 paid_search 1 0.418
7 87 paid_search 0 0.418
8 88 paid_search 0 0.418
10 90 paid_search 1 0.418
12 92 paid_search 0 0.418
13 93 paid_search 1 0.418
14 94 paid_search 1 0.418
22 102 paid_search 1 0.418
23 103 paid_search 1 0.418
1 81 mobile 1 0.254
2 82 mobile 1 0.254
9 89 mobile 0 0.254
11 91 mobile 0 0.254
16 96 mobile 1 0.254
20 100 mobile 0 0.254
25 105 mobile 1 0.254
4 84 affiliate 1 0.209
6 86 affiliate 0 0.209
18 98 affiliate 1 0.209
19 99 affiliate 0 0.209
21 101 affiliate 0 0.209
5 85 social_media 1 0.119
15 95 social_media 1 0.119
17 97 social_media 1 0.119
24 104 social_media 1 0.119By taking a quick look, we immediately notice that the first two consumers did not make purchase (i.e., user_purchase=0). However, the model predicts that they both have the highest conversion probability among all consumers (44.8%).
Knowing that the first channel exposure for both consumers is paid search, we can explain the discrepancy between actual and predicted data in the following way. The model estimation using the training set gives us high transition probability from paid search to conversion whereas in the test set consumers do not behave homogeneously with those in the training set.
Now let’s count the cumulative number of actual conversions based on the ordered data.
```{r}
first_touch_prediction$cumu_conversion_ft <- cumsum(first_touch_prediction$user_purchase)
# take a look at the first five rows of the data with cumulative count
first_touch_prediction[1:5,]
``` user_id first_touch user_purchase predicted_conversion_probability
3 83 paid_search 1 0.418
7 87 paid_search 0 0.418
8 88 paid_search 0 0.418
10 90 paid_search 1 0.418
12 92 paid_search 0 0.418
cumu_conversion_ft
3 1
7 1
8 1
10 2
12 2We can see that among the first five customers with highest predicted conversion probabilities, there are only two customers (users id: 83 and 90) who actually made purchases. This means that using the top 20% valuable customers (in terms of conversion potential) based on our prediction, we are able to capture only two out of 16 total conversions in the test set. This coressponds to 12.5% of actual purchases.
Question: Do you think this is a satisfactory performance? Why or why not? (Hint: think of what the output would look like if we made predictions via a randomly selected data.)
Last-touch model
We follow the same steps as we did above with the first-touch model.
```{r}
last_touch_prediction <- last_touch[order(-last_touch$predicted_conversion_probability),]
last_touch_prediction
last_touch_prediction$cumu_conversion_lt <- cumsum(last_touch_prediction$user_purchase)
# take a look at the first five rows of the data with cumulative count
last_touch_prediction[1:5,]
``` user_id user_purchase predicted_conversion_probability
4 84 1 0.269
1 81 1 0.119
2 82 1 0.119
3 83 1 0.119
5 85 1 0.119
6 86 0 0.119
7 87 0 0.119
8 88 0 0.119
9 89 0 0.119
10 90 1 0.119
11 91 0 0.119
12 92 0 0.119
13 93 1 0.119
14 94 1 0.119
15 95 1 0.119
16 96 1 0.119
17 97 1 0.119
18 98 1 0.119
19 99 0 0.119
20 100 0 0.119
21 101 0 0.119
22 102 1 0.119
23 103 1 0.119
24 104 1 0.119
25 105 1 0.119
user_id user_purchase predicted_conversion_probability cumu_conversion_lt
4 84 1 0.269 1
1 81 1 0.119 2
2 82 1 0.119 3
3 83 1 0.119 4
5 85 1 0.119 5Next, we look at the Markov model predictions.
Markov model
As with first- and last-touch models, we need to calculate the cumulative actual conversions that Markov model is able to capture.
```{r}
markov_prediction$cumu_conversion_mm <- cumsum(markov_prediction$user_purchase)
# take a look at the first five rows of the data with cumulative count
markov_prediction[1:5,]
``` users_id user_purchase predicted_conversion_probability cumu_conversion_mm
1 81 1 0 1
2 82 1 0 2
3 83 1 0 3
4 85 1 0 4
5 86 0 0 4From this output, it seems that Markov chain performs much better than the first-touch model. For the first five customers with highest predicted conversion probabilities, all of them have actually made purchases in real life.
Question: Can you comment on the prediction performance of Markov model vs. first-touch model?
7.3 Model performance visualization
Finally, it is always a good idea to further visualize the results on models’ predictive power so that Levi and Emma can make more informed decisions.
As described above, we can plot the cumulative actual conversions captured by each model onto one graph.
In the following code, Cumu_Markov and Cumu_First_Touch refer to cumulative number of actual conversions captured by Markov and first-touch predictions, respectively. Observation identifies the observation with \(n^{th}\) (n=1,2,…, 25) highest predicted conversion probability.
```{r}
# prepare a dataset for plotting
plot_raw <- data.frame(Observation= 1:25, Cumu_Markov = markov_prediction$cumu_conversion_mm, Cumu_First_Touch = first_touch_prediction$cumu_conversion_ft, Cumu_Last_Touch = last_touch_prediction$cumu_conversion_lt)
plot_raw
``` Observation Cumu_Markov Cumu_First_Touch Cumu_Last_Touch
1 1 1 1 1
2 2 2 1 2
3 3 3 1 3
4 4 4 2 4
5 5 4 2 5
6 6 4 3 5
7 7 4 4 5
8 8 4 5 5
9 9 5 6 5
10 10 5 7 6
11 11 5 8 6
12 12 6 8 6
13 13 7 8 7
14 14 8 9 8
15 15 9 9 9
16 16 10 10 10
17 17 11 11 11
18 18 11 11 12
19 19 11 12 12
20 20 12 12 12
21 21 13 12 12
22 22 14 13 13
23 23 15 14 14
24 24 16 15 15
25 25 16 16 16```{r}
plot_markov <- c(plot_raw$Cumu_Markov[5]/16,plot_raw$Cumu_Markov[10]/16,plot_raw$Cumu_Markov[15]/16,plot_raw$Cumu_Markov[20]/16,plot_raw$Cumu_Markov[25]/16)
plot_first_touch <- c(plot_raw$Cumu_First_Touch[5]/16,plot_raw$Cumu_First_Touch[10]/16, plot_raw$Cumu_First_Touch[15]/16, plot_raw$Cumu_First_Touch[20]/16, plot_raw$Cumu_First_Touch[25]/16)
plot_last_touch <- c(plot_raw$Cumu_Last_Touch[5]/16,plot_raw$Cumu_Last_Touch[10]/16, plot_raw$Cumu_Last_Touch[15]/16, plot_raw$Cumu_Last_Touch[20]/16, plot_raw$Cumu_Last_Touch[25]/16)
plot <- data.frame(Markov = plot_markov, First_Touch = plot_first_touch,Last_Touch = plot_last_touch, Observation = c(5,10,15,20,25)/25 )
ggplot(plot,aes(x=Observation))+
geom_line(aes(y=Markov), color = "darkred") +
geom_line(aes(y=First_Touch), color="blue") +
geom_line(aes(y=Last_Touch), color="orange")+
xlab('% of Observations') +
ylab('Cumulative % of Actual Conversion Captured') +
scale_color_discrete(name = "Model Approach", labels = c("Markov", "First Touch"))
```
On the above graph,
The dark red line represents Markov model while the blue line refers to the first-touch model, and the orange line refers to the last-touch model.
The horizontal axis refers to the percentage of ranked observations. We have 25 predicted observations in total. The first 20% means the first five observations with highest predicted conversion probability.
The vertical axis refers to the percentage of actual conversions captured by our model using a specific % of ranked observations. For example, a \(Y\)% capture with \(X\)% observations means that our model is able to capture \(Y\)% of actual sales by using the first \(X\)% of ranked observations.
The graph above further shows that Markov model (dark red line) always outperforms first- and last-touch model (blue and orange line, respectively) in capturing actual conversions. For example, within the first 20% of observations (i.e., the first 5 observations with highest likelihood of conversion), Markov model is able to capture 25% of the actual conversions (i.e., 4 out of 16) already while the first-touch model can only capture 12.5% (i.e., 2 out of 16). One can argue that Markov chain model relatively does a better job but its predictive accuracy is not high. Tweaking the Markov model further (e.g., trying higher order models) with a larger and more balanced dataset, we can better understand how much can be gained from this model in practice.
Footnotes
In this R application, the words ‘channels’ and ‘touchpoints’ are used interchangeably↩︎